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BACKGROUND OF THE INVENTION 

Field of the Invention 

The present invention relates to embedding data in material. Embodiments of the 
invention relate to watermarking material. 

"Material" as used herein means information material represented by information 
signals which includes at least one or more of image material, audio material and data 
material. Image material is generic to still and moving images and includes video and other 
information signals representing images. 

Description of the Prior Art 

Steganography is the embedding of data into material such as video material, audio 
material and data material in such a way that the data is imperceptible in the material. 

Data may be embedded as a watermark in material such as video material, audio 
material and data material. A watermark may be imperceptible or perceptible in the material. 

A watermark may be used for various purposes. It is known to use watermarks for the 
purpose of protecting the material against, or trace, infringement of the intellectual property 
rights of the owner(s) of the material. For example a watermark may identify the owner of 
the material. 

Watermarks may be "robust" in that they are difficult to remove from the material. 
Robust watermarks are useful to trace the provenance of material which is processed in some 
way either in an attempt to remove the mark or to effect legitimate processing such as video 
editing or compression for storage and/or transmission. Watermarks may be "fragile" in that 
they are easily damaged by processing which is useful to detect attempts to remove the mark 
or process the material. 

Visible watermarks are useful to allow e.g. a customer to view an image e.g. over the 
Internet to determine whether they wish to buy it but without allowing the customer access to 
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the unmarked image they would buy. The watermark degrades the image and the mark is 
preferably not removable by the customer. Visible watermarks are also used to determine the 
provenance of the material into which they are embedded. 

It is known to embed a watermark into material by applying a spatial frequency 
transform to the material embedding the watermark in the spatial frequency transform and 
applying an inverse transform to the watermarked material. A scaling factor is applied to the 
watermark. It is desirable to choose a scaling factor to improve the ability of the watermark 
to withstand unauthorised attempts to remove it; allow efficient authorised removal; reduce 
degradation of the unmarked material; and ensure that the mark is imperceptible where an 
imperceptible mark is desired. Those properties may be incompatible. Also, when material 
has been watermarked, it is desirable to be able to remove the mark. However, embedding a 
watermark in the material in such a way as to make difficult unauthorised removal may also 
have the consequence that the watermark is difficult to remove by an authorised person. 

SUMMARY OF THE INVENTION 

According to a first aspect of the present invention, there is provided a method of 
embedding data in material, the method comprising the steps of: 

producing transform coefficients Ci-Ci_representing a spatial frequency transform of 
the material, and 

combining the coefficients CjGi with bits RjRi of the data to produce modified 
coefficients Ci ! Q 1 where 

Ci'-Ci + Vi Ri Q' =G ± cxi Ri 

the method further comprising determining otj¥i for each unmodified coefficient CjGi 
as a function F{Cnii F(Cn)i of a predetermined set (Cnli fCnli of transform coefficients C^Gn 
which set excludes the coefficient CjGi wherein the coefficients are serially ordered and the 
coefficients C n Gft are coefficients preceding coefficient CjGi. 
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Preferably, the set (C n |i fCnli of transform coefficients is: 

a) a set consisting of unmodified coefficients; or 

b) a set consisting of modified coefficients; or 

c) a set comprising modified and unmodified coefficients. 

Thus oti^i is adapted to each coefficient to which it is applied, allowing it to minimise 
degradation of the material. That also allows ccj to make the embedded data more robust 
against processing which intentionally or unintentionally damages the embedded data. 

The set IC n H -fG^H-of coefficients used to calculate otjVi associated with coefficient 
CiCi excludes CjGi. As will become apparent from the method of removing the data RjRi, 
that allows exact recalculation of aiVi in the removal process and thus exact removal of RjRi 
to restore the original material if no processing has occurred, and no clipping of the image in 
the spatial domain has occurred. 

The invention allows cxj¥i to be related to the other coefficients from which it is 
calculated by any suitable function. 

The transform may produce coefficients Cj€i in a plurality of frequency bands. The 
transform coefficients forming the set (Cnl (C n i ; may be all in the same band. The transform 
coefficients forming the set ICn ii fCnli may be in a plurality of bands. Using a set of 
coefficients (Cnli fCnli in a plurality of bands allows the data RjRi to be concealed in the 
material using material properties in bands other than the band containing the data RjRi. In a 
preferred embodiment, the coefficients are serially ordered and the coefficients C n Gn are 
unmodified coefficients preceding coefficient CjGi. During removal of the embedded data 
such ordering allows the coefficients to be used to calculate Vi for a subsequent coefficient 
CiGi. 

In such circumstances, the set ICn ii fCnli may be: 

a) the set consisting of unmodified coefficients; or 

b) a set consisting of modified coefficients; or 
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c) a set comprising modified and unmodified coefficients. 

Also, according to the first aspect of the present invention there is provided a method 
of removing data embedded in material according to the method of said one aspect, the 
method comprising the steps of. 

determining the values of bits RjRi of the data; 

calculating, for each modified coefficient CilGi 1 , the value of the said function FICnii 
F (Cn) i of the corresponding set (C n >i fCn ) i of coefficients CnGft to determine ctiVi; and 

for each modified coefficient C^Ci 1 , subtracting therefrom ai.RiVirRi to restore the 
unmodified coefficient value-GiCi, wherein the coefficients are serially ordered and the 
coefficients Cn are coefficients preceding coefficient CjGi. 

In a preferred embodiment, otjVi is calculated from a set (Cnli fCnli of unmodified 
coefficients. The method thus uses the restored coefficient CjGi as an unmodified coefficient 
C E Gft of another set (C n ij fCnl i-of unmodified coefficients for restoring another coefficient 
CjlGj 1 . It will be appreciated that the set (Cni j 4C**4-i-excludes the coefficient CjlCi 2 . The set 
(Cnli ("Cn]i is of unmodified coefficients allowing ocjYi to be calculated exactly from the 
material in which the data RjRi is embedded. As a modified coefficient CjlCi 1 is restored to 
its original value it is then available to be used to calculate Vj-Vj_for another coefficient 

In a preferred embodiment; the coefficients are serially ordered and the coefficients 
C a €n are unmodified coefficients preceding coefficient CjGi. During removal of the 
embedded data such ordering allows the coefficients to be used to calculate VjVi for a 
subsequent coefficient j 

In such circumstances, the set fCnli fCn)i may be: 

a) the set consisting of unmodified coefficients; or 

b) a set consisting of modified coefficients; or 

c) a set comprising modified and unmodified coefficients. 
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According to a second aspect of the invention there is provided: 

a) A method of removing data embedded in material comprising the steps of 
receiving material in which data is embedded; 

accessing an information store storing information enabling the data to be 
removed; and 

removing the said data using the enabling data accessed from the store. 

b) A method comprising the steps of 
embedding data in material; and 

storing in an information store information for enabling the data to be removed 
from the material. 

c) Apparatus for removing data embedded in material comprising: 
an input for receiving material in which data is embedded; 

an information store for storing information enabling the data to be removed; 

and 

a remover arranged to remove the said data using the enabling data accessed 
from the store. 

d) Apparatus comprising: 

an embedder for embedding data in material; 

a store for storing information for enabling the data to be removed from the 
material; and 

a generator for generating the enabling information when the said data is 
embedded in the material. 

The provision of the stored enabling data allows access to enabling data which allows 
the embedded data to be removed. In preferred embodiments of this aspect of the invention, 
the enabling data is stored in a manner which is secure against unauthorised access to it. 

Methods of preventing unauthorised access to secure data are well known. 
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It has also been found that, in for example video material, embedding watermarks in 

identical fashion in different images results in differing degrees of difficulty in removing the 
watermarks. 

According to the second aspect of the present invention, there is also provided a 
method comprising the steps of: 

embedding data in first material to produce second material in which data is 
embedded; 

removing the data from the second material to produce recovered material; 

comparing the first and recovered material to determine the differences and locations 
of differences therebetween; and 

storing corrections which correct the said differences and data identifying the said 
locations at which the differences occur in the first material. 

The second aspect also provides apparatus for embedding data in material comprising: 

an embedder for embedding data in first material to produce second material in which 
data is embedded; 

a remover for removing the data from the second material to produce recovered 
material; 

a comparator for comparing the first and recovered material to determine the 
differences and locations of differences therebetween; and 

a store for storing data identifying the said locations and corrections which correct the 
said differences. 

According to the second aspect of the present invention there is also provided a 
method of removing data embedded in material, the data being embedded in the material 
according to the embedding method of said second aspect, the removing method comprising 
the steps of: 
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removing the data from the second material to produce recovered material; deriving 

the said corrections and locations from a store storing data identifying the said locations and 
corrections which correct the said differences; and 

using the corrections to correct the recovered material at the said locations. 

The second aspect also provides apparatus for removing data from material in which 
the data has been embedded by the apparatus of said second aspect, the removing apparatus 
comprising: 

a remover for removing the data from the second material to produce recovered 
material; 

a deriver for deriving the said corrections and locations from the said store; and 
a corrector arranged to use the stored corrections to correct the recovered material at 
the said locations. 

The second aspect of the invention also provides a system comprising a combination 
of the apparatus of the said second aspect (herein after a data embedder) and the apparatus of 
said third aspect (herein after a data remover). 

Ideally, the second aspects of the invention operate optimally with a lossless channel 
between the embedding apparatus and the removing apparatus. If the channel between the 
embedding apparatus and the removing apparatus is known, an emulator emulating that 
channel may be provided in the embedding apparatus between the embedded and remover of 
the embedding apparatus. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The above and other objects, features and advantages of the invention will be apparent 

from the following detailed description of illustrative embodiments which is to be reads in 

connection with the accompanying drawings, in which: 

Figure 1 is a schematic block diagram of a watermark embedding and removal system; 
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Figure 2 is a more detailed schematic block diagram of an embedder of the system of 

Figure 1 : 

Figures 3 A and B illustrate an example of a window of coefficients and how the 
window relates to a coefficient Gi-C^being modified to embed a bit of a watermark; 

Figure 4 is a flow diagram of a method of calculating strength V in accordance with 
an example of the invention; 

Figure 5 is a schematic block diagram of a watermark decoder; 

Figure 6 is a schematic block diagram of a watermark remover: 

Figure 7 is a flow diagram of a method of calculating strength V in accordance with 
an example of the invention; 

Figure 8 is a schematic diagram of an alternative, illustrative, set of coefficients usable 
to calculate V: 

Figure 9 is a schematic diagram illustrating the operation of frame stores in the 
embedder of Figure 2 and the remover of Figure 6; 

Figure 10 is a schematic block diagram of an illustrative watermark embedding and 
removal system in accordance with the invention; 

Figure 1 1 is a schematic block diagram of an embedder of the system of Figure 10; 

Figure 12 is a schematic block diagram of a remover of the system of Figure 10; 

Figures 13 and 14 are diagrams used herein below to describe wavelets transforms; 

and 

Figures 15 and 16 are diagrams of data structures of UMIDs. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
Figure 1 illustrates a watermarking system for embedding, recovering and removing a 
watermark onto or from a video image L The watermarking system 10 comprises a source 
1 10 of the image 1, a strength adapter 180, a watermark embedder 120, a watermark decoder 
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140, a watermark remover 130 and a store 1 50. The decoder and remover may be coupled to 
the embedder via a channel 125 which may include a video processor, and/or a store. 

In overview, the watermark embedder 120 embeds a watermark onto a video image I 
to form a watermarked image T, the watermark decoder 140 recovers the watermark from the 
watermarked image I 1 and the watermark remover 130 removes the watermark from the 
watermarked image I 1 to produce a restored image I". The restored image I" may not be 
identical to the original image 1, especially if the channel 125 includes a processor and/or if 
clipping of the image in the spatial domain occurs. 

The watermark embedder 120 receives, in this example, as watermark data, a UMID. 
UMMDs are described in the section UMIDs below. The strength adapter 180 determines the 
magnitude of a parameter a, referred to herein as the strength of the watermark in relation to 
the video image I. The strength V is determined such that the watermark may be recovered 
whilst minimising its perceptibility to a viewer of the watermarked image I'. The 
watermarked image I' may then be stored, and/or transmitted and/or routed for further 
processing, in the channel 125. 

The watermark decoder 140 generates a restored UMID 145 from the watermarked 
image I\ The watermark remover 130 generates a restored image I" from the watermarked 
image I' using the restored UMID. 
Watermark embedder. Figure 2. 

Figure 2 illustrates the watermark embedder 120 in more detail. The watermark 
embedder 120 comprises pseudo-random sequence generator 220, an error correction coding 
generator 200, a wavelet transformer 210, an inverse wavelet transformer 250, a first 
combiner 230, a data converter 225 and a second combiner 240. The wavelet transformer 210 
includes a frame store FS1. The inverse transformer 250 includes a frame store FS2. The 
frame store FS1 stores a frame of unmodified coefficients Ci. The frame store FS2 stores a 
frame of modified coefficients-Gi 1 CY. 
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The error correction coding generator 200 receives the UMID and outputs an error 

correction coded UMID to the first combiner 230. The pseudo-random sequence generator 
220 outputs a pseudo-random binary sequence (PRBS)-PitJPi; where i is the j th bit of the 
sequence, to the first combiner 230. The PRBS has a length L x J of bits where J is the 
number of bits in the error correction encoded UMID. Each bit j of the error correction 
encoded UMID then modulates a section of length L of the PRBS. The first combiner 230 
logically combines the error correction encoded UMID with the PRBS to produce a 
watermark having bits RjRi. A bit Wj-Q Wj =Q of the error correction encoded UMID 
inverts L bits of the PRBS. A bit Wj~l Wj=l of the error correction encoded UMID does not 
invert the PRBS. Thus bits WjWf of the error correction encoded UMID are spread over L 
bits of the PRBS. The data converter 225 converts binary 1 to symbol +1 and binary 0 to 
symbol -1 to ensure that binary 0 bits contribute to a correlation value used in the decoder of 
Figure 5. 

The wavelet transformer 210 receives the video image I from the source 110 and 
outputs wavelet coefficients CiCi to the second combiner 240. Wavelets are briefly discussed 
in the section Wavelets below. 

The second combiner 240 receives the watermark-fe Ru the wavelet coefficients CjCi 
and watermark strength Vi and outputs modified coefficients Gi^Cj! where 

Ci'- Ci + Vi Ri C V = Cj + oci Ri 

The inverse wavelet transformer 250 receives the modified coefficients C^Gi 1 and 
outputs a spatial domain watermarked image I\ 

The embedder includes an ECC generator 200. The use of error correction coding to 
produce an error correction coded UMID is advantageous since it allows the UMID 1 75 to be 
reconstructed more readily should some information be lost. This provides a degree of 
robustness to future processing or attacks against the watermark. The use of a pseudo- 
random sequence Pi-Pi_to generate a spread spectrum signal for use as a watermark is 
advantageous since it allows the error correction coded UMID 205 to be spread across a large 
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number of bits. Also, it allows the watermark to be more effectively hidden and reduces the 
visibility of the watermark. Applying the watermark to a wavelet transform of the image is 
advantageous since this reduces the perceptibility of the watermark. Furthermore, the 
strength of the watermark is adjusted by VjVi to ensure that the watermark is not perceptible. 

The operation of the error correction code generator 200 will now be described. The 
error correction code generator 200, receives a UMID. Typically the UMID will be a binary 
sequence of 3 1 bytes. The error correction code generator 200 typically outputs a 5 1 1 bit 
error correction coded binary sequence. Various error correction coding schemes are known. 
One approach uses BCH coding which corrects up to 31 bit errors. The error correction rates 
can be further improved by using knowledge of the UMID format to help correct errors. One 
such approach is to check for invalid dates times GPS locations etc. 

The watermark is preferably embedded in predetermined regions of the wavelet 
transformed image. Most preferably the upper horizontal (hH,l V)and upper vertical (lH.hV) 
bands are used. These bands are chosen as watermarks embedded in these regions are not 
readily perceptible. The length of the pseudo-random sequence may be chosen such that the 
watermark fills the predetermined regions in each wavelet image. The regions in which the 
watermark is embedded may be within a border of unmodified coefficients thereby allowing 
the image to the spatially shifted without the watermark being lost. 

Calculating V, Figures 3 and 4. 

In accordance with an illustrative embodiment of the invention; for each coefficient-Ci 
Cj, a value of V, Vi-V^is calculated VjVi is calculated as 

V i- F f C n h,a i=F (C a > u 

where (Cn)i fCnh is a set of unmodified wavelet coefficients excluding CjGi, which 
set may vary with i, that is respective values of ctiVi are functions F of respective sets 
(Cnli t'CnU , This is shown as step S8 in Figure 4. 



11 



Attorney Docket No. 282430US 

Inventors: Jonathan J. STONE, et al. 

Serial No. 10/006294 
Filed: December 6, 2001 

Filed with Reply to Office Action dated: September 26, 2005 

SUBSTITUTE SPECIFICATION 

Marked-up Copy 

The coefficients (C n ii fCnli of the set may be in the same wavelet band as Ci_<Si-or 

may be in different bands from CjCi and from each other as described below with reference to 
Figure 8. 

If the coefficients are in the same band as CjGi, they are preferably in a window 
adjacent CjGi. For example the set comprises N coefficients C i-1 to Q-N Ci 1 to Ci N as 
shown in Figure 3 and the embodiment will be described in the following with reference to 
that. 

The number N of coefficients may vary with CjGi; thus for generality N is denoted as 

QNi. 

The function F may be any suitable function. In this illustrative embodiment F is such 

that 

ai ~ F [d H-^ jp VX! C n for n ~ i 1 tQ * N for N **° and ai ~ k for Ni ~ °- 

a, =F(Cn >/ = — JYC n 2 for n = i-1 to i-N for Afafl and = k for N i = 0. 
Ni v 

Figure 3 A is a map of wavelet coefficients in a frame store 300, the coefficients being 
in level 1 of a wavelet transform. In a preferred embodiment, the coefficients CjGi are 
modified only in the upper horizontal hH, IV and upper vertical 1H, hV bands to embed the 
watermark. However, coefficients in other bands and/or in other levels may be modified to 
embed a watermark. In the following only band hH, IV is considered. 

The wavelet coefficients are stored in the frame store 300 (also denoted FS1 in Figure 
2) and in this example are stored as shown in Figure 3A grouped in the bands. The 
coefficients are serially ordered. For example they may be serially ordered by a raster scan 
thereof. Other scanning patterns are known. Assuming serial ordering of the coefficients in 
each band, for each coefficient CjGi to be modified, there is defined a set (C n li (Cn)i ; (herein 
also referred to as a "window 1 ) of NiNj coefficients excluding CjGi. The set (Cnii fCnji 
consists of the NjNi-coefficients Ci-M to C i-Ni Ci 1 to Ci Ni preceding coefficient CjGi on the 
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same line, up to a maximum of for example M most recent coefficients. It will be noted that 
in the band hH,l V coefficient Gl-Cjjias no preceding coefficients. G2-C?has only one 
preceding coefficient, and so on. For coefficient Cu Vi CI Vi is set to a predetermined value 
K. For subsequent coefficients the set comprises the totality of preceding coefficients. 

Thus VjVi is defined individually for each coefficient Cj€i to be modified. In the 
example above it is defined by the set of NjNi unmodified coefficients preceding CjGi. By 
choice of the appropriate function F, VjYi is adapted to the image such that image degradation 
can be minimised. In addition as will be discussed below in the section Remover, this allows 
Vj_Vi to be recalculated from the watermarked image coefficients, after those have been 
restored to their original values. This improves the accuracy of restoring the original image. 

Referring to Figure 4 the illustrative procedure for calculating Cj'is as follows :- 
The calculation procedure starts at step S2. At step S4, i is initialised with value 0. At step 
S6, i is incremented by 1 to calculate VI at step S8 for coefficient QiGir. At step S 10 the 
value of modified coefficient C^C^ is calculated. The procedure then reverts to step S6 and i 
is incremented. The procedure continues until all coefficients have been modified. 

In addition, the calculation of VjV4- may be modified in one or both of the following 

ways:- 

lj If a i <a T i^ it is incremented to pc t t, where octt_ is a lower threshold; and 

if oc i > pi th it is reduced to oc t h, where oc th is an upper threshold. 

2) The magnitude I C n I of each coefficient is compared with a threshold C th. 

If I Cn 1 >C th then Cn is not included in the calculation of cx i ; or 

if 1 C n I >C TH jhen Cn is clipped to (CJ |Cn |)CTH_ 

If Vi^V rp^T- it is increm e nt e d to ^q^-where v-qi^ is a lower threshold; and 

tf^p^-v-^- it is r e duced to Vqm^ wh e r e v^n -is an upper thr e shold. 

3) Th e magnitud e *Cn* of e ach co e fficient is compar e d with a thr e shold c m 

If *C ft*>GrH th e n C B is not included in th e calculation of Vi,; er 
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tf*€tt*^qrH then C » is clipped to (C n /*Cn*)C w 

Watermark decoder and remover. Figures 5 and 6. 
Decoder Figure 5 

The operation of the watermark decoder 140 will now be explained in more detail 
with reference to Figure 5. The watermark decoder 140 receives the watermarked image I' 
and outputs the restored UMID. The watermark decoder 140 comprises a wavelet 
transformer 310, a reference pseudo-random sequence (PRBS) generator 320, a correlator 
330, a selector 340 and a error correction coding decoder 350. The PRBS generated by the 
generator 320 is identical to that generated by the PRBS generator 220 of Figure 2 and 
converted by a data converter (not shown ) to values +1 and -1 as described above. 

The wavelet transformer 310 receives the watermarked image I' and, in known 
manner, outputs the modified wavelet coefficients QiGi 1 . The correlator 330 receives the 
reference pseudo-random sequence PRBS having symbols Pi of values +1 and -1 from the 
pseudo-random sequence generator 320, and the wavelet coefficients C^Gi 1 and outputs a 
watermark image bit correlation sequence 335. The watermarked image bit correlation 
sequence is determined in the following way. 

The modified wavelet coefficients Ci'=CH-a iRi where Ri Ci'-Ci+ViRi where Ri are 
bits of PRBS modulated by error-correction encoded bits Wj-W^of UMID. In the example 
given above there are 5 1 1 bits W^Wf. Each bit WjWj- modulates L bits of PRBS. There are 
JL bits in the modulated PRBS. 

For each error correction encoded bit WjWj, the correlater 330 calculates a correlation 

value 

JL+L jL+l 

S'j ° £ t Ci , J>i_ S' J = ^Ci'.Pi 

/= jL+1 i=jL+\ 
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where j =0, 1, 2... T-l, and T is the number of error correction encoded bits. In this 

example T=51 1 . A sequence 335 of correlation values S'j is produced. 

The correlation sequence 335 is received by the selector 340 which outputs an 
uncorrected UMID 345. The selector 340 outputs a bit value "1" for a value of S f greater than 
0 and a bit value "0" for S' less than or equal to 0. The error correction code decoder 350 
receives the uncorrected UMID 345 and in known manner outputs the restored UMID 145. 

The reference PRBS Pi-Pi_is synchronised with the modulated PRBS in the 
watermarked image. For that purpose a synchroniser (not shown) is used. Such 
synchronisation is known in the art. 
Remover Figure 6. 

The watermark remover 130 receives the restored UMID 145, and the watermarked 
image I 1 and outputs a restored image I". The watermark remover 130 comprises a pseudo- 
random sequence generator 420 for generating a reference pseudo random sequence PjPi 
identical to that produced by generators 220 and 320, a spread spectrum signal generator 430 
which produces, via a data converter 425, a restored watermark RTRi4iaving bit values +1 
and -1 from the restored UMID 145 and the pseudo-random sequence PiPi. The reference 
sequence PiPi is synchronised with the modulated sequence in the watermarked image in 
known manner. 

The watermark remover 130 further comprises a wavelet transformer 410 which 
produces modified wavelet coefficients CjlGi 1 from the watermarked image I', a strength 
estimator 460 for calculating V4-Vj_and a combiner 440 which calculates restored wavelet 
coefficient values according to the equation 

Ci-Ci' V i-Ri ^ Q = C[ - a,. Ri\ 

The restored wavelet coefficients C are fed to an inverse wavelet transformer 450 
which outputs the restored image I". 

Calculating V i Vi, Figure 7. 
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In accordance with the illustrative embodiment of the invention, VjVi is calculated in 

the embedder as described above in the section Calculating The estimator 460 of the 
remover of Figure 6 recalculates a in analogous manner from coefficients Cj€i which have 
been restored to their original values. 

Thus referring for example to Figure 3 A and to Figures 6 and 7, the modified 
coefficients C\IG¥ are stored in a frame store 300 indicatd as FS3 in the wavelet transformer 
of Figure 6 in the same way as shown in figure 3 A and they are serially ordered in the same 
way as described with reference to Figure 3 A. It will be recalled that coefficient CjlCi 1 has no 
preceding coefficients so Vi = k and CI = CI' kRl . = k and = CV- kRu For each 
subsequent coefficient CjGi, VjVi is calculable from the set of NjNi of preceding restored 
coefficients, all of which have been restored to their original value according to 
Ci-Ci 1 V ir-Ri^ Ci = Ci* - a,. R i\ 

Referring to Figure 7, the calculation procedure starts at step S5. At step S7, i is 
initialised to 0. At step S9, i is incremented by 1 to calculate a\Vi at step SI 1 for coefficient 
QiGV. At step S13 the original value CiG4- is calculated from coefficients CilG^. The 
procedure then reverts to step S9 and i is incremented. The procedure continues until all 
coefficients C^G* 1 have been restored to their original values C|Gi. 

As in the embedder of Figure 2, the calculation of a may be modified in one or both of 
the following ways:- 

1) If oc i <a T j_, it is incremented to pc t t, where oc t i_ is a lower threshold; and 

if a, i > oc th it is reduced to oc t h, where oct h is an upper threshold. 

2) The magnitude I C» 1 of each coefficient is compared with a threshold C th. 

If I C n I >C th then Cn is not included in the calculation of eg; or 

if I C n 1 >C th . then Cn is clipped to (Cn /|Cn|)OrH- 

4-) If Vi<Vq^, it is incr e m e nt e d to V^, wh e r e is a low e r thr e shold, and 

if Vi> Vi pH- it is r e duc e d to V m/ wher e V^is an upp e r thr e shold. 
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2-) Tho magnitud e Cn^C^H, of each coeffici e nt is compar e d with a thr e shold C th- 

If *C H * >Cq m . th e n C „ is not included in th e calculation of Vi; or 
->€qiH- > th e n C ft is clipp e d to (C f/^MSf^GqpHT 

Modifications. 

As mentioned above the coefficients from which the value of (ctjeei is calculated may 
be in different bands to the related coefficient CjGi which is to be modified or restored to its 
original value. Thus by way of example, referring to Figure 8; the set of coefficients (Cnli 
-{€n}-i-used to calculate oci¥i of band hH, IV may be in the other bands. In the example of 
Figure 8 the set {C n )i [Cnli is shown as including coefficients Cli, C2i and C3i Cn, and 
C3i_which are at positions related to the position of coefficient CjGi. In this way, image 
properties in other bands are taken into account in calculating otj¥i to ensure that the 
watermark is imperceptible. 

The coefficients Cli, C2i and C3i C iu C? i and used to modify or restore CjCi, may 
be coefficient which are never modified. That can be done by modifying only coefficients in 
one or more bands such as hH, IV and leaving the coefficients in other bands unmodified. 
Alternatively at least some of the coefficients CI i, C2i and C3i Cn , and Cii used to 
modify or restore C\Gi may be modified. That can be done by storing the coefficients in a 
frame store 300 as shown in Figure 3 or 8 and by reading out coefficients in an order which 
allows the procedures of Figures 4 and 7 to be followed. 

It will be appreciated that whilst the foregoing discussion refers for ease of 
explanation to only 3 coefficients Cli, C2i and C3i C u, and C^i in 3 bands in one level, in 
practice many more coefficients may be used and the coefficients may be in more than three 
bands and in more than one level. 
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Other transforms 

Whilst the invention has been described by way of example with reference to Wavelet 
transforms, it may be used with other transforms for example DCT. 

Other material 

Whilst the invention has been described by way of example with reference to material 
comprising video material ( still or moving images), it may be applied to other material, for 
example audio material and data material. 

PRBS 

As described hereinabove, the PRBS has a length of L J where J is the number of bits 
in a UMID. Thus each bit ^-W^of the UMID modulates a section of length L of the PRBS. 
Instead, it may have a length of L bits and be repeated for each bit j of the UMID. 

Other Watermark data 

Whilst the invention has been described by way of example with reference to UMIDs 
as the watermark data, it may be used with other data as the watermark. 

Using modified coefficients to calculate ViV ; 

The foregoing embodiment calculates VjV-i using unmodified coefficients. In 
alternative embodiments V is calculated using modified coefficients or a combination of 
modified and unmodified coefficients. The coefficients CjC* are serially ordered. The 
coefficients used to calculate Vi for coefficient CjGi are coefficients preceding i on the serial 
order. 

Referring to Figures 2, 6 and 9 frames stores FS1, FS2, FS3 and FS4 are provided in 
the wavelet transformer 210, the inverse wavelet transformer 250, the wavelet transformer 
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410 and the inverse wavelet transformer 450. Frame stores FS 1 and FS4 store unmodified 
coefficients. Frame stores FS2 and FS3 store modified coefficients CiGi, 

Thus there are available both at the encoder and at the remover serially ordered sets of 
unmodified and modified coefficients. 

In the embedder of Figure 2, as coefficients CiGi in store FS1 are modified, they are 
stored in FS2 as coefficients C/Gi 1 . Thus modified coefficients C^Gi^-are available to 
calculate VjYi. Thus the set (Cn l/ f Cnli used to calculate Vj¥i for modifying coefficient C,Gi 
may comprise modified coefficients C preceding C,Gi optionally together with unmodified 
coefficients C preceding C,Gi. 

At the remover modified coefficients C^Gi 1 are stored in store FS3. As the 
coefficients are restored, restored coefficients Cj_Gi-are stored in store FS4. Thus modified 
coefficients C are available to calculate Vj¥i optionally together with restored coefficients C. 

As diagrammatically shown in Figure 9, sets of coefficients preceding a coefficient C\_ 
or Q; Ci or Ci' are present in all four frame stores FS1, FS2, FS3 and FS4. 

Shape of sets (C n }/ {Cn)i 

A set IC n }/ fCnli may have any convenient shape. Where VjVi is calculated only 
from coefficients preceding CiGi, the set may consist of coefficients immediately preceding 
CjGi. Where the coefficients are raster scanned to serially order them, the set may consist of 
coefficients on the same scanning line as CjGi. Alternatively, it may consist of coefficients on 
that line and a preceding line. Other shapes are possible. 

Figure 10 illustrates a system, in this example a watermarking system, for embedding 
a watermark onto an image I and for recovering and removing it from the image. The 
watermarking system comprises: a source 1 10 of the image I; a sub system A for applying the 
watermark; a sub-system B for detecting and removing the watermark; a channel C linking 
the sub-systems A and B; and a database D. 
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In overview, the subsystem A applies the watermark to the image. The watermarked 

image is transmitted via the channel C to the subsystem B where the watermark is detected 
and removed. It is desired to restore the original image in the subsystem B with no 
degradation. For that purpose, at the subsystem A the watermark is removed from the image 
by a remover 130 and the thus restored image is compared with the original image I to detect 
differences and the locations within the image of the differences. The database D stores the 
locations of differences and correction data which may be values of the original image at 
those locations or the differences. The subsystem B detects and removes the watermark to 
produce a substantially restored image. The removal process is the same as at the subsystem 
A. The locations of corrections and the corrections are read from the database and the 
corrections applied to the restored image to correct it. In practice the system will operate on 
many images. Thus it is necessary to identify the images and the correction data associated 
therewith in the database D. In this example each image is identified by an identifier which is 
used to access the relevant data in the database. Preferably the identifier is a UMID. UMIDs 
are described in the section UMIDs below. Most preferably the watermark comprises the 
UMID. In the example of Figure 1, the image is transformed by a wavelet transform. 
Wavelet transforms are discussed in the section Wavelets below. In the example of Figure 1 
wavelet coefficients are compared to determine the said differences. 

First Example 

Referring to Figure 10, the source 110 produces a spatial domain image I. An 
embedder 120 receives a UMID from a generator 115 and embeds the UMID as the 
watermark in wavelet coefficients CjGi of a wavelet transform (T) of the image according to 

Ci' ~ Ci + V.Ri CV = Ci ± a.Ri 

where CjG* is the i'" original image coefficient. C\G^ is the i th modified coefficient, Rj_ 
Ri-is i th bit of watermark data and a is a scaling factor. As described hereinafter RjRi may be 
a bit of a pseudo random symbol sequence modulated by the UMID data WirWj. 
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The remover 130 removes the watermark to produce restored coefficients C^Gi^ 

according to 

Ci" - Ci' V.Ri G" = Cj! - a.R^ 
A comparator 125 compares the restored coefficients C^Gi 11 with the original 
coefficients CjGi to determine any differences and the locations thereof. The differences and 
locations i are stored in the database D as correction data together with the UMID generated 
by generator 115. 

The coefficients C^Gi 1 produced by the embedder 120 are inverse transformed (T* 1 ) 
and applied to the channel C. 

The subsystem B receives the watermarked image from the channel C. A transformer 
T recreates the coefficients Qi_G^. The UMID is detected and removed by a detector arid 
remover 127. The detected UMID is used to access the correction data in the data base and 
the correction data is applied to the image coefficients C^Gi^in a corrector 129 to restore the 
original coefficients CjGi which may be inverse transformed ( T _1 )to restore the original 
image 1 . 

In this first example, a is a fixed predetermined value. Using a fixed predetermined 
value of a facilitates removal of the watermark. 

For the purposes of the first example, the channel C is preferably "lossless": that is it 
does not distort the watermarked spatial domain image. 

Second Example 

The second example is identical to the first except that a channel emulator 121 is 
provided between the embedder 120 and remover 130. The emulator applies, to the output of 
the embedder, a channel emulator function emulating the effect of channel C on the output of 
the embedder. 

The channel emulation 121 emulates the channel C. Thus errors introduced by the 
channel C can be detected and corrections stored in the database D. 
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This is useful especially if the channel C is lossy. 

Third Example 

This modifies the first or second example in that V is not fixed. The embedder 120 is 
shown in Figure 1 1 and the remover 130, 127 is shown in Figure 12. 

Figure 1 1 is identical to Figure 2 described above except that it has a connection 
supplying the wavelet coefficients Cj to the comparator 125 of Figure 10. Likewise, Figure 
12 is identical to Figure 6 above except it has a connection supplying the restored coefficients 
to the comparator 125 of Figure 10. Thus no further description is needed of Figures 1 1 and 
12. 

Modifications 

Whilst the aspect of the invention described with reference to Figures 10 to 12 embeds 
and removes watermarks as described with reference to Figures 1 to 9, other watermark 
embedding and removal techniques may be used. 

Wavelets 

Wavelets are well known and are described in for example 66 A Really Friendly Guide 
to Wavelets " by C Valens, 1999 and available at 
http://perso.wanadoo.fr/polyvalens/clemens/wavelets/wavelets.html. 

Valens shows that the discrete wavelet transform can be implemented as an iterated 
filter bank as used in sub-band coding, with scaling of the image by a factor of 2 at each 
iteration. 

Thus referring to Figure 13, a spatial domain image is applied to a set of high pass HP 
and low pass LP filters. At level 1, the first stage of filtering, the image is filtered 
horizontally and vertically and, in each direction, scaled down by a factor of 2. In level 2, the 
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low pass image from level 1 is filtered and scaled in the same way as in level 1. The filtering 
and scaling may be repeated in subsequent levels 3 onwards. 

The result is shown schematically in Figure 14. Figure 10 is a representation normal 
in the art. At level one the image is spatially filtered into four bands: the lower horizontal and 
vertical band, lHi, hVi; the upper horizontal band hHi, lVi: the upper vertical band lHj, hVi; 
and the upper horizontal and vertical band, hHi, hVj. At level 2, the lower horizontal and 
vertical band. lHi, 1 Vi is filtered and scaled into the lower horizontal and vertical band, 1H 2 , 
IV2: the upper horizontal band hVi; 1 V 2 : the upper vertical band 1H 2 , bV 2 ; and the upper 
horizontal and vertical band, hH 2 , hV 2 . At level 3 (not shown in Figure 10), the lower 
horizontal and vertical band. 1H 2 . 1 V 2 is further filtered and scaled. 

UMIDs 

The UMID or Unique Material Identifier is described in SMPTE Journal March 2000. 
Referring to Figure 15 an extended UMID is shown. It comprises a first set of 32 bytes of 
basic UMID and a second set of 32 bytes of signature metadata. 

The first set of 32 bytes is the basic UMID. The components are: 

• A 12-byte Universal Label to identify this as a SMPTE UMID. It defines the type of 
material which the UMID identifies and also defines the methods by which the globally 
unique Material and locally unique Instance numbers are created. 

• A 1 -byte length value to define the length of the remaining part of the UMID. 

• A 3-byte Instance number which is used to distinguish between different 'instances 1 
of material with the same Material number. 

• A 16-byte Material number which is used to identify each clip. Each Material 
number is the same for related instances of the same material. 

The second set of 32 bytes of the signature metadata as a set of packed metadata items 
used to create an extended UMID. The extended UMID comprises the basic UMID followed 
immediately by signature metadata which comprises: 
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• An 8-byte time/date code identifying the time and date of the Content Unit creation. 

• A 12-byte value which defines the spatial co-ordinates at the time of Content Unit 
creation. 

• 3 groups of 4-byte codes which register the country, organisation and user codes 
Each component of the basic and extended UMIDs will now be defined in turn. 
The 12-byte Universal Label 

The first 12 bytes of the UMED provide identification of the UMID by the registered 



string value defined in table 1 . 



Byte No. 


Description 


Value (hex) 


1 


Object Identifier 


06h 


2 


Label size 


OCh * | 


3 


Designation: ISO 


2Bh 


4 


Designation: SMPTE 


34h 


5 


Registry: Dictionaries 


Olh 


6 


Registry: Metadata Dictionaries 


Olh 


7 


Standard: Dictionary Number 


Olh 


8 


Version number 


Olh 


9 


Class: Identification and location 


Olh 


10 


Sub-class: Globally Unique Identifiers 


Olh 


11 


Type: UMID (Picture, Audio, Data, Group) 


01,02, 03, 04h 


12 


Type: Number creation method 


XXh 



Table 1: Specification of the UMID Universal Label 

The hex values in table 1 may be changed: the values given are examples. Also the 
bytes 1-12 may have designations other than those shown by way of example in the table. 
Referring to the Table 1, in the example shown byte 4 indicates that bytes 5-12 relate to a data 
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format agreed by SMPTE. Byte 5 indicates that bytes 6 to 10 relate to "dictionary" data. 
Byte 6 indicates that such data is "metadata" defined by bytes 7 to 10. Byte 7 indicates the 
part of the dictionary containing metadata defined by bytes 9 and 10. Byte 10 indicates the 
version of the dictionary. Byte 9 indicates the class of data and Byte 10 indicates a particular 
item in the class. 

In the present embodiment bytes 1 to 10 have fixed preassigned values. Byte 11 is variable. 
Thus referring to Figure 15, and to Table 1 above, it will be noted that the bytes 1 to 10 of the 
label of the UMID are fixed. Therefore as shown in Figure 16 they may be replaced by a 1 
byte 'Type' code T representing the bytes 1 to 10. The type code T is followed by a length 
code L. That is followed by 2 bytes, one of which is byte 1 1 of Table 1 and the other of 
which is byte 12 of Table 1, an instance number (3 bytes) and a material number (16 bytes). 
Optionally the material number may be followed by the signature metadata of the extended 
UMID and/or other metadata. 

The UMID type (byte 11) has 4 separate values to identify each of 4 different data 
types as follows: 

'Olh' = UMID for Picture material 

'02h' = UMID for Audio material 

'03h' = UMID for Data material 

'04h' = UMID for Group material (i.e. a combination of related essence). 

The last (12 th ) byte of the 12 byte label identifies the methods by which the material 
and instance numbers are created. This byte is divided into top and bottom nibbles where the 
top nibble defines the method of Material number creation and the bottom nibble defines the 
method of Instance number creation. 

Length 

The Length is a 1-byte number with the value '13h' for basic UMIDs and c 33h' for 
extended UMIDs. 

Instance Number 
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The Instance number is a unique 3 -byte number which is created by one of several 
means defined by the standard. It provides the link between a particular 'instance' of a clip 
and externally associated metadata. Without this instance number, all material could be 
linked to any instance of the material and its associated metadata. 

The creation of a new clip requires the creation of a new Material number together 
with a zero Instance number. Therefore, a non-zero Instance number indicates that the 
associated clip is not the source material. An Instance number is primarily used to identify 
associated metadata related to any particular instance of a clip. 

Material Number 

The 16-byte Material number is a non-zero number created by one of several means 
identified in the standard. The number is dependent on a 6-byte registered port ID number, 
time and a random number generator. 

Signature Metadata 

Any component from the signature metadata may be null-filled where no meaningful 
value can be entered. Any null-filled component is wholly null-filled to clearly indicate a 
downstream decoder that the component is not valid. 

The Time-Date Format 

The date-time format is 8 bytes where the first 4 bytes are a UTC (Universal Time 
Code) based time component. The time is defined either by an AES3 32-bit audio sample 
clock or SMPTE 12M depending on the essence type. 

The second 4 bytes define the date based on the Modified Julian Data (MJD) as 
defined in SMPTE 309M. This counts up to 999,999 days after midnight on the 17 th 
November 1858 and allows dates to the year 4597. 

The Spatial Co-ordinate Format 

The spatial co-ordinate value consists of three components defined as follows: 
• Altitude: 8 decimal numbers specifying up to 99,999,999 metres. 
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• Longitude: 8 decimal numbers specifying East/West 180.00000 degrees (5 decimal 
places active). 

• Latitude: 8 decimal numbers specifying North/South 90.00000 degrees (5 decimal 
places active). 

The Altitude value is expressed as a value in metres from the centre of the earth thus 
allowing altitudes below the sea level. 

It should be noted that although spatial co-ordinates are static for most clips, this is 
not true for all cases. Material captured from a moving source such as a camera mounted on a 
vehicle may show changing spatial co-ordinate values. 
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Country Code 

The Country code is an abbreviated 4-byte alpha-numeric string according to the set 
defined in ISO 3166. Countries which are not registered can obtain a registered alpha- 
numeric string from the SMPTE Registration Authority. 

Organisation Code 

The Organisation code is an abbreviated 4-byte alpha-numeric string registered with 
SMPTE. Organisation codes have meaning only in relation to their registered Country code 
so that Organisation codes can have the same value in different countries. 

User Code 

The User code is a 4-byte alpha-numeric string assigned locally by each organisation 
and is not globally registered. User codes are defined in relation to their registered 
Organisation and Country codes so that User codes may have the same value in different 
organisations and countries. 

Although illustrative embodiments of the invention have been described in detail 
herein with reference to the accompanying drawings, it is to be understood that the invention 
is not limited to those precise embodiments, and that various changes and modifications can 
be effected therein by one skilled in the art without departing from the scope and spirit of the 
invention as defined by the appended claims. 
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